Driving assistance device, driving assistance method, and storage medium

ABSTRACT

According to an embodiment, a driving assistance device includes a recognizer configured to recognize a surrounding situation of a vehicle, a driving controller configured to control at least steering of the vehicle on the basis of the surrounding situation recognized, a receiver configured to receive an operation of an occupant of the vehicle, and an output controller configured to cause an output to output a traveling state of the vehicle. The driving controller causes a lane change of the vehicle to be made when a receiver receives an instruction for allowing the vehicle to make the lane change and a condition in which the lane change is made is satisfied. The output controller causes the output to output information about the lane change when the lane change has not been started even though a first prescribed time period has elapsed after the receiver received the instruction.

CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2021-062233,filed Mar. 31, 2021, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a driving assistance device, a drivingassistance method, and a storage medium.

Description of Related Art

In recent years, research on automated driving for automaticallycontrolling the traveling of vehicles has been conducted. In thisregard, technology for outputting a command for changing a travelinglane of a host vehicle from a first lane to a second lane to a drivingcontroller for controlling automated driving when an operation of a userto move a host vehicle object within an image displayed on a displayfrom the first lane to the second lane is received is known (forexample, PCT International Publication No. WO 2017/022197).

SUMMARY

Incidentally, when a situation in which a lane change cannot be starteddue to an influence of a surrounding situation or the like continuesafter an occupant issues a lane change instruction, the occupant mayforget that he or she has issued the lane change instruction. Thus, ifthe lane change is made after a while from the issuance of the lanechange instruction, the lane change may be made in a state in which theoccupant is not expecting the lane change and appropriate drivingassistance may not be provided.

Aspects of the present invention have been made in consideration of suchcircumstances and an objective of the present invention is to provide adriving assistance device, a driving assistance method, and a storagemedium capable of providing more appropriate driving assistance.

A driving assistance device, a driving assistance method, and a storagemedium according to the present invention adopt the followingconfigurations.

(1): According to an aspect of the present invention, there is provideda driving assistance device including: a recognizer configured torecognize a surrounding situation of a vehicle; a driving controllerconfigured to control at least steering of the vehicle on the basis ofthe surrounding situation recognized by the recognizer; a receiverconfigured to receive an operation of an occupant of the vehicle; and anoutput controller configured to cause an output to output a travelingstate of the vehicle, wherein the driving controller causes a lanechange of the vehicle to be made when a receiver receives an instructionfor allowing the vehicle to make the lane change and a condition inwhich the lane change is made is satisfied, and wherein the outputcontroller causes the output to output information about the lane changewhen the lane change has not been started by the driving controller eventhough a first prescribed time period has elapsed after the receiverreceived the instruction.

(2): In the above-described aspect (1), the information about the lanechange includes information about cancelation of the lane change.

(3): In the above-described aspect (1), the output controller causes aprocess of outputting the information to end when the receiver has notreceived an instruction for canceling the lane change even though asecond prescribed time period has elapsed after the information aboutthe lane change was output to the output, and the driving controllercauses a process of making the lane change to continue.

(4): In the above-described aspect (3), the second prescribed timeperiod is shorter than the first prescribed time period.

(5): In the above-described aspect (1), the output includes a displayand the output controller causes the display to display an image showinginformation about the lane change.

(6): In the above-described aspect (3), the output controller causes theoutput to output information about the lane change when the lane changehas not been started by the driving controller even though a thirdprescribed time period has elapsed after the process of outputting theinformation was ended.

(7): In the above-described aspect (6), the third prescribed time periodis a time period equal to the first prescribed time period or a timeperiod that is changed in accordance with the number of times theinformation about the lane change has been output or the time when theinformation about the lane change has been output.

(8): According to an aspect of the present invention, there is provideda driving assistance method including: recognizing, by a computer, asurrounding situation of a vehicle; controlling, by the computer, atleast steering of the vehicle on the basis of the recognized surroundingsituation in a driving controller; receiving, by the computer, anoperation of an occupant of the vehicle; causing, by the computer, anoutput to output a traveling state of the vehicle; causing, by thecomputer, a lane change of the vehicle to be made when an instructionfor allowing the vehicle to make the lane change is received and acondition in which the lane change is made is satisfied; and causing, bythe computer, the output to output information about the lane changewhen the lane change has not been started by the driving controller eventhough a first prescribed time period has elapsed after the instructionwas received.

(9): According to an aspect of the present invention, there is provideda computer-readable non-transitory storage medium storing a program forcausing a computer to: recognize a surrounding situation of a vehicle;control at least steering of the vehicle on the basis of the recognizedsurrounding situation in a driving controller; receive an operation ofan occupant of the vehicle; cause an output to output a traveling stateof the vehicle; cause a lane change of the vehicle to be made when aninstruction for allowing the vehicle to make the lane change is receivedand a condition in which the lane change is made is satisfied; and causethe output to output information about the lane change when the lanechange has not been started by the driving controller even though afirst prescribed time period has elapsed after the instruction wasreceived.

According to the above-described aspects (1) to (9), it is possible toprovide more appropriate driving assistance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of a vehicle system using a drivingassistance device according to an embodiment.

FIG. 2 is a functional configuration diagram of a first controller and asecond controller.

FIG. 3 is a diagram showing an example of relationships between drivingmodes, control states of a vehicle, and tasks.

FIG. 4 is a diagram for describing a lane change determiner, an approvaldeterminer, and an HMI controller.

FIG. 5 is a diagram showing an example of a lane change suggestionimage.

FIG. 6 is a diagram showing an example of an image showing informationabout a lane change.

FIG. 7 is a diagram for describing a case where a lane change has notbeen canceled by the approval determiner.

FIG. 8 is a flowchart showing an example of a flow of a process executedby an automated driving controller of the embodiment.

DESCRIPTION OF EMBODIMENTS

Embodiments of a driving assistance device, a driving assistance method,and a storage medium of the present invention will be described belowwith reference to the drawings. Hereinafter, an embodiment in which thedriving assistance device is applied to an automated driving vehiclewill be described as an example. For example, automated driving is aprocess of executing driving control by automatically controlling one orall of steering, acceleration, and deceleration of the vehicle. Thedriving control of the vehicle may include, for example, various typesof driving assistance control such as adaptive cruise control (ACC),auto lane changing (ALC), a lane keeping assistance system (LKAS), andtraffic jam pilot (TJP). Driving of the automated driving vehicle may becontrolled according to manual driving of an occupant (a driver).

[Overall Configuration]

FIG. 1 is a configuration diagram of a vehicle system 1 using thedriving assistance device according to an embodiment. A vehicle in whichthe vehicle system 1 is mounted is, for example, a vehicle such as atwo-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle,and a drive source thereof is an internal combustion engine such as adiesel engine or a gasoline engine, an electric motor, or a combinationthereof. The electric motor operates using electric power generated by apower generator connected to the internal combustion engine or electricpower when a secondary battery or a fuel cell is discharged.

For example, the vehicle system 1 includes a camera 10, a radar device12, a light detection and ranging sensor (LIDAR) 14, an objectrecognition device 16, a communication device 20, a human machineinterface (HMI) 30, a vehicle sensor 40, an operator 45, a navigationdevice 50, a map positioning unit (MPU) 60, a driver monitor camera 70,driving operation elements 80, an automated driving controller 100, atravel driving force output device 200, a brake device 210, and asteering device 220. Such devices and equipment are connected to eachother by a multiplex communication line such as a controller areanetwork (CAN) communication line, a serial communication line, or awireless communication network. The configuration shown in FIG. 1 ismerely an example and some of the components may be omitted or othercomponents may be further added. The HMI 30 is an example of an“output.” The operator 45 is an example of a “receiver.” The operator 45and the automated driving controller 100 is an example of a “drivingassistance device.”

For example, the camera 10 is a digital camera using a solid-stateimaging element such as a charge coupled device (CCD) or a complementarymetal oxide semiconductor (CMOS). The camera 10 is attached to anyposition on the vehicle (hereinafter, a vehicle M) in which the vehiclesystem 1 is mounted. When the view in front of the vehicle M is imaged,the camera 10 is attached to an upper part of a front windshield, a rearsurface of a rearview mirror, or the like. For example, the camera 10periodically and iteratively images the surroundings of the vehicle M.The camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves aroundthe vehicle M and detects at least a position (a distance to and adirection) of a physical object by detecting radio waves (reflectedwaves) reflected by the physical object. The radar device 12 is attachedto any position on the vehicle M. The radar device 12 may detect aposition and a speed of the physical object in a frequency modulatedcontinuous wave (FM-CW) scheme.

The LIDAR 14 radiates light (or electromagnetic waves of a wavelengthclose to an optical wavelength) to the vicinity of the vehicle M andmeasures scattered light. The LIDAR 14 detects a distance to an objecton the basis of a time period from light emission to light reception.The radiated light is, for example, pulsed laser light. The LIDAR 14 isattached to any position on the vehicle M.

The object recognition device 16 performs a sensor fusion process ondetection results from some or all of the camera 10, the radar device12, and the LIDAR 14 to recognize a position, a type, a speed, and thelike of a physical object. The object recognition device 16 outputsrecognition results to the automated driving controller 100. The objectrecognition device 16 may output detection results of the camera 10, theradar device 12, and the LIDAR 14 to the automated driving controller100 as they are. The object recognition device 16 may be omitted fromthe vehicle system 1.

The communication device 20 communicates with another vehicle in thevicinity of the vehicle M using, for example, a cellular network, aWi-Fi network, Bluetooth (registered trademark), dedicated short rangecommunication (DSRC), or the like, or communicates with various types ofserver devices via a radio base station.

The HMI 30 outputs various types of information to the occupant of thevehicle M under the control of the HMI controller 170. The HMI 30 mayfunction as a receiver that receives an input operation by the occupant.The HMI 30 includes, for example, a display 32 and a speaker 34. The HMI30 may include a microphone, a buzzer, a key, an indicator lamp and thelike.

The display 32 is, for example, a liquid crystal display (LCD), anorganic electro-luminescence (EL) display, or the like. The display 32is provided, for example, near the front of the driver's seat (the seatclosest to the steering wheel) on an instrument panel, and is installedat a position where the occupant (specifically, the driver) can performvisual recognition from the steering wheel gap or through the steeringwheel.

Information necessary for traveling during manual driving or automateddriving of the vehicle M (hereinafter referred to as driving assistanceinformation) is displayed as an image on the display 32. The drivingassistance information includes, for example, information such as aspeed of the vehicle M, an engine speed, the remaining amount of fuel, aradiator water temperature, a traveling distance, a state of a shiftlever, a lane (a marking) or other vehicles recognized by the objectrecognition device 16, the automated driving controller 100, or thelike, a lane in which the vehicle M should travel, and a future targettrajectory. The driving assistance information may include informationabout switching of a driving mode to be described below, informationabout a traveling state based on driving assistance, information about alane change based on the driving assistance (information about thesuggestion of the lane change and the cancelation of the lane change),and the like.

The display 32 may be installed near the center of the instrument panelin addition to the above-described position. In this case, in additionto the driving assistance information, the display 32 displays, forexample, an image showing a navigation result of the navigation device50 and the like. The display 32 may display a television program or maydisplay content stored in a DVD or content such as a movie downloadedfrom an external device via the communication device 20.

The display 32 may include, for example, a head-up display (HUD). TheHUD, for example, projects an image onto a part of the front windshieldin front of the driver's seat so that the eyes of the occupant sittingin the driver's seat can see the virtual image. For example, the drivingassistance information or the like is displayed on the HUD. The display32 may be configured as a touch panel having a function of a receiverthat receives an operation input from the occupant.

At least one speaker 34 is installed within a cabin. For example, thespeaker 34 outputs speech, a warning sound, or the like under thecontrol of the HMI controller 170.

The vehicle sensor 40 includes a vehicle speed sensor configured todetect the speed of the vehicle M, an acceleration sensor configured todetect acceleration, a yaw rate sensor configured to detect an angularspeed around a vertical axis, a direction sensor configured to detect adirection of the vehicle M, and the like. The vehicle sensor 40 mayinclude a position sensor that acquires a position of the vehicle M. Theposition sensor is, for example, a sensor that acquires positioninformation (longitude/latitude information) from a Global PositioningSystem (GPS) device. The position sensor may be a sensor that acquiresposition information using a global navigation satellite system (GNSS)receiver 51 of the navigation device 50.

The operator 45 receives an operation of the occupant of the vehicle M.The operator 45 includes, for example, a mode changeover switch thatreceives an operation of switching the driving mode to be describedbelow, an approval switch that approves the suggestion of the lanechange under the control of the system side (the automated drivingcontroller 100 side), and a cancelation switch that cancels theexecution of the approved lane change, and the like. The approval switchand the cancelation switch may be included in one switch. In this case,for example, the approval and cancellation operations can be performedwhen the switch is turned on and off. For the operator 45, for example,a mechanical switch is attached to a steering wheel, an instrumentpanel, or the like. The operator 45 is configured integrally with theHMI 30, and for example, the above-mentioned switch may be displayed asa graphical user interface (GUI) switch on the display 32.

For example, the navigation device 50 includes the GNSS receiver 51, anavigation HMI 52, and a route determiner 53. The navigation device 50retains first map information 54 in a storage device such as a hard diskdrive (HDD) or a flash memory. The GNSS receiver 51 identifies aposition of the vehicle M on the basis of a signal received from a GNSSsatellite. The position of the vehicle M may be identified or correctedby an inertial navigation system (INS) using an output of the vehiclesensor 40. The navigation HMI 52 includes a display, a speaker, a touchpanel, keys, and the like. The navigation HMI 52 may be partly or whollyshared with the above-described HMI 30. For example, the routedeterminer 53 determines a route (hereinafter referred to as a route ona map) from the position of the vehicle M identified by the GNSSreceiver 51 (or any input position) to a destination input by theoccupant using the navigation HMI 52 with reference to the first mapinformation 54. The first map information 54 is, for example,information in which a road shape is expressed by a link indicating aroad and nodes connected by the link. The first map information 54 mayinclude curvature of a road, point of interest (POI) information, andthe like. The route on the map is output to the MPU 60. The navigationdevice 50 may perform route guidance using the navigation HMI 52 basedon the route on the map. The navigation device 50 may be implemented,for example, according to a function of a terminal device such as asmartphone or a tablet terminal possessed by the occupant. Thenavigation device 50 may transmit a current position and a destinationto a navigation server via the communication device 20 and acquire aroute equivalent to the route on the map from the navigation server.

For example, the MPU 60 includes a recommended lane determiner 61 andstores second map information 62 in a storage device such as an HDD or aflash memory. The recommended lane determiner 61 divides the route onthe map provided from the navigation device 50 into a plurality ofblocks (for example, divides the route every 100 [m] in a travelingdirection of the vehicle), and determines a recommended lane for eachblock with reference to the second map information 62. For example, therecommended lane determiner 61 determines in what lane numbered from theleft the vehicle will travel when a road on which the vehicle M iscurrently traveling or will travel in the near future has a plurality oflanes. The recommended lane determiner 61 determines the recommendedlane so that the vehicle M can travel along a reasonable route fortraveling to a branching destination when there is a branch point in theroute on the map.

The second map information 62 is map information which has higheraccuracy than the first map information 54. For example, the second mapinformation 62 includes information about a center of a lane,information about a boundary of a lane, and the like. The second mapinformation 62 may include road information (a type of road), the numberof lanes of the road, the presence or absence of a branch point or amerge point, legal speeds (a speed limit, a maximum speed, and a minimumspeed), traffic regulation information, address information (anaddress/postal code), facility information, telephone numberinformation, and the like. The second map information 62 may be updatedat any time when the communication device 20 communicates with anotherdevice.

The driver monitor camera 70 is, for example, a digital camera that usesa solid-state image sensor such as a CCD or a CMOS. For example, thedriver monitor camera 70 is attached to any location on the vehicle Mwith respect to a position and a direction where the head of an occupant(hereinafter referred to as a driver) sitting in the driver's seat ofthe vehicle M can be imaged from the front (in a direction in whichhis/her face is imaged). For example, the driver monitor camera 70 isattached to an upper part of a display provided on the central portionof the instrument panel of the vehicle M.

The driving operation elements 80 include, for example, an acceleratorpedal, a brake pedal, a shift lever, and other operation elements inaddition to the steering wheel 82. A sensor for detecting an amount ofoperation or the presence or absence of an operation is attached to thedriving operation element 80 and a detection result is output to theautomated driving controller 100 or some or all of the travel drivingforce output device 200, the brake device 210, and the steering device220. The steering wheel 82 is an example of an “operation element forreceiving a steering operation by the driver.” The operation elementdoes not necessarily have to be annular and may be in the form of avariant steering wheel, a joystick, a button, or the like. A steeringgrip sensor 84 is attached to the steering wheel 82. The steering gripsensor 84 is implemented by a capacitance sensor or the like, andoutputs a signal for detecting whether or not the driver is gripping thesteering wheel 82 (indicating that the driver is in contact with thesteering wheel 82 in a state in which a force is applied) to theautomated driving controller 100.

The automated driving controller 100 includes, for example, a firstcontroller 120, a second controller 160, an HMI controller 170, and astorage 180. Each of the first controller 120, the second controller160, and the HMI controller 170 is implemented, for example, by ahardware processor such as a central processing unit (CPU) executing aprogram (software). Some or all of the above components may beimplemented by hardware (including a circuit; circuitry) such as alarge-scale integration (LSI) circuit, an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), or agraphics processing unit (GPU) or may be implemented by software andhardware in cooperation. The program may be prestored in a storagedevice (a storage device including a non-transitory storage medium) suchas an HDD or a flash memory of the automated driving controller 100 ormay be stored in a removable storage medium such as a DVD or a CD-ROMand installed in the HDD or the flash memory of the automated drivingcontroller 100 when the storage medium (the non-transitory storagemedium) is mounted in a drive device. A combination of the action plangenerator 140 and the second controller 160 is an example of a “drivingcontroller.” The HMI controller 170 is an example of an “outputcontroller.”

The storage 180 may be implemented by the above-described various typesof storage devices or a solid-state drive (SSD), an electricallyerasable programmable read-only memory (EEPROM), a read-only memory(ROM), a random-access memory (RAM), or the like. The storage 180stores, for example, information, a program, various other types ofinformation, and the like necessary for providing the driving assistancein the present embodiment.

FIG. 2 is a functional configuration diagram of the first controller 120and the second controller 160. The first controller 120 includes, forexample, a recognizer 130, the action plan generator 140, and a modedeterminer 150. For example, the first controller 120 implements afunction based on artificial intelligence (AI) and a function based on apreviously given model in parallel. For example, an “intersectionrecognition” function may be implemented by executing intersectionrecognition based on deep learning or the like and recognition based onpreviously given conditions (signals, road markings, or the like withwhich pattern matching is possible) in parallel and performingcomprehensive evaluation by assigning scores to both the recognitions.Thereby, the reliability of automated driving is secured.

The recognizer 130 recognizes states of positions (or relativepositions), speeds (or relative speeds), acceleration, and the like ofphysical objects (for example, other vehicles or other obstacles) nearthe vehicle M on the basis of information input from the camera 10, theradar device 12, and the LIDAR 14 via the object recognition device 16.For example, the position of the physical object is recognized as aposition on absolute coordinates with a representative point (a centerof gravity, a driving shaft center, or the like) of the vehicle M as theorigin and is used for control. The position of the physical object maybe represented by a representative point such as a center of gravity ora corner of the physical object or may be represented by a representedarea. The “state” of a physical object may include acceleration or jerkof the physical object or an “action state” (for example, whether or nota lane change is being made or intended) when the physical object is amoving object such as another vehicle.

For example, the recognizer 130 recognizes a lane in which the vehicle Mis traveling (a traveling lane). For example, the recognizer 130recognizes the traveling lane by comparing a pattern of a road marking(for example, an arrangement of solid lines and broken lines) obtainedfrom the second map information 62 with a pattern of road markings inthe vicinity of the vehicle M recognized from an image captured by thecamera 10. The recognizer 130 may recognize the traveling lane byrecognizing a traveling path boundary (a road boundary) including a roadmarking, a road shoulder, a curbstone, a median strip, a guardrail, orthe like as well as a road marking. In this recognition, a position ofthe vehicle M acquired from the navigation device 50 or a processingresult of the INS may be added. The recognizer 130 recognizes atemporary stop line, an obstacle, red traffic light, a toll gate, a roadsign, and other road events. The recognizer 130 recognizes an adjacentlane adjacent to the traveling lane. The adjacent lane is, for example,a lane in which traveling is possible in the same direction as thetraveling lane.

When the traveling lane is recognized, the recognizer 130 recognizes aposition or an orientation of the vehicle M with respect to thetraveling lane. For example, the recognizer 130 may recognize a gap of areference point of the vehicle M from the center of the lane and anangle formed with respect to a line connected to the center of the lanein a traveling direction of the vehicle M as a relative position and anorientation of the vehicle M related to the traveling lane.Alternatively, the recognizer 130 may recognize a position of thereference point of the vehicle M related to one side end (a road markingor a road boundary) of the traveling lane or the like as a relativeposition of the vehicle M related to the traveling lane. Here, thereference point of the vehicle M may be the center of the vehicle M orthe center of gravity. The reference point may be an end (a front end ora rear end) of the vehicle M or may be a position where one of aplurality of wheels provided in the vehicle M is present.

The action plan generator 140 generates a future target trajectory alongwhich the vehicle M automatedly travels (independently of the driver'soperation) so that the vehicle M can generally travel in the recommendedlane determined by the recommended lane determiner 61 and cope with asurrounding situation of the vehicle M. For example, the targettrajectory includes a speed element. For example, the target trajectoryis represented by sequentially arranging points (trajectory points) atwhich the vehicle M is required to arrive. The trajectory points arepoints at which the vehicle M is required to arrive for each prescribedtraveling distance (for example, about several meters [m]) along a road.In addition, a target speed and target acceleration for each prescribedsampling time (for example, about several tenths of a second [sec]) aregenerated as parts of the target trajectory. The trajectory point may bea position at which the vehicle M is required to arrive at the samplingtime for each prescribed sampling time. In this case, information aboutthe target speed or the target acceleration is represented by aninterval between the trajectory points. When a set speed of the vehicleM is predetermined, the action plan generator 140 may generate a targettrajectory so that the speed of the vehicle M becomes the set speedwithin a range in which the vehicle M can travel.

The action plan generator 140 may set an automated driving event(function) when a target trajectory is generated. Automated drivingevents include a constant-speed traveling event, a low-speed trackingevent, a lane change event, a branch point-related movement event, amerge point-related movement event, a takeover event, and the like. Theaction plan generator 140 generates a target trajectory according to anactivated event. When the action plan generator 140 executes the drivingcontrol of the vehicle M, a prescribed event, or the like, the actionplan generator 140 may suggest (recommend) the driving control and theexecution of the event to the occupant in accordance with a driving modeof the vehicle M to be described below and generate a correspondingtarget trajectory if the suggestion has been approved.

The action plan generator 140 includes, for example, a lane changedeterminer 142 and an approval determiner 144. Details of functions ofthese components will be described below.

The mode determiner 150 sets the driving mode executed by the vehicle Mto any one of a plurality of driving modes in which tasks imposed on theoccupant are different (in other words, a plurality of modes havingdifferent degrees of automation) on the basis of a situation of thevehicle M and the like. The driving controller causes the vehicle M totravel in the mode determined by the mode determiner 150. The modedeterminer 150 includes, for example, a driver state determiner 152 anda mode change processor 154.

FIG. 3 is a diagram showing an example of relationships between drivingmodes, control states of the vehicle M, and tasks.

The driving modes of the vehicle M include, for example, five modes frommode A to mode E. A degree of automation of the control state, i.e., thedriving control of the vehicle M, is highest in mode A among modes A toE, decreases in the order of mode B, mode C, and mode D, and is lowestin mode E. In contrast, the task imposed on the occupant is mildest inmode A, becomes severer in the order of mode B, mode C, and mode D, andis severest in mode E in which manual driving is performed. In modes Bto E, the control state is not automated driving, so the automateddriving controller 100 is responsible for ending the control related toautomated driving and performing the shift to driving assistance ormanual driving. Modes A and B are examples of a “first driving mode” andmode C is an example of a “second driving mode.” Hereinafter, details ofeach mode for the driver that is an example of the occupant will bedescribed.

In mode A, the state is an automated driving state and any one of tasksof monitoring the surroundings of the vehicle M and gripping thesteering wheel 82 (hereinafter referred to as a “steering grip”) is notimposed on the driver. It is determined whether or not the driver ismonitoring the surroundings on the basis of, for example, a capturedimage analysis result of the driver monitor camera 70 and it isdetermined whether or not the driver is gripping the steering accordingto, for example, a detection result of the steering grip sensor 84.Surrounding monitoring includes at least monitoring in front of thevehicle M. The front is a space in the traveling direction of thevehicle M that is visually recognized via a front windshield. However,even in mode A, the driver is required to be in a posture in which afast shift to manual driving is enabled in response to a request fromthe system centered on the automated driving controller 100. The term“automated driving” mentioned herein means that both the steering andspeed of the vehicle M are controlled without depending on the driver'soperation.

For example, mode A is a driving mode that is executable when acondition in which the vehicle M is traveling at a prescribed speed (forexample, about 50 [km/h]) or less on a motorway such as an expresswayand there is a preceding vehicle to be tracked is satisfied and may bereferred to as a “TJP mode.” When the condition for traveling in mode Ais no longer satisfied or when the mode switching operation has beenreceived by the operator 45, the mode determiner 150 changes the drivingmode of the vehicle M to another mode (for example, mode B).

The driver can execute a second task while mode A is being executed. Thesecond task is, for example, an act other than the driver's drivingpermitted during the automated driving of the vehicle M. Second tasksinclude, for example, watching television, making portable phone calls,sending and receiving emails, eating, and the like.

In mode B, the state is a driving assistance state and a task ofmonitoring the surroundings of the vehicle M (hereinafter, surroundingmonitoring) is imposed on the driver, but the task of gripping thesteering wheel 82 is not imposed on the driver. In mode B, for example,ACC, ALC, LKAS, and the like are executed. ACC, ALC, and LKAS may alsobe executed, for example, in modes C and D under the restriction of aprescribed task. For example, in mode B, the lane change (ALC) of thevehicle M based on a process in which the navigation device 50 sets aroute to a destination or the like is made according to thedetermination of the vehicle system 1 side in a state in which the lanechange instruction is not received from the driver. The lane change isused to move the vehicle M from a host vehicle lane in which the vehicleM is traveling to an adjacent lane adjacent to the host vehicle lane.

In mode C, the state is a driving assistance state and the task ofmonitoring surroundings and the task of gripping the steering wheel 82are imposed on the driver. In mode C, for example, a suggestion is sentto the driver via the HMI 30 when it is determined that the lane changeof the vehicle M is required on the vehicle system 1 side and the lanechange is made according to driving control for controlling at least thesteering among the steering, the acceleration, and the deceleration ofthe vehicle M when the approval of the lane change by the driver hasbeen received from the operator 45.

In the case of mode C, even in a state in which the lane change cannotbe made the present time point, if it is determined that it ispreferable to make the lane change on the vehicle system 1 side in thefuture, a suggestion may be sent to the driver and the lane changeaccording to the driving control may be made when the approval(reservation) of the lane change is received and then an executioncondition of the lane change of the vehicle M is satisfied on the basisof a surrounding situation and the like. The execution condition of thelane change is, for example, a condition in which the action plangenerator 140 generates a target trajectory for making the lane changefrom the traveling lane of the vehicle M to a lane that is a lane changedestination. For example, when there is another vehicle in the travelinglane of the vehicle M or the lane that is the lane change destination,the action plan generator 140 sets a target position serving as the lanechange destination and generates a target trajectory for moving to thetarget position without contact with the other vehicle on the basis of arelative position and/or a relative distance between the vehicle M andthe other vehicle. When the target position cannot be set due to trafficsituations such as traffic jams or construction work, the action plangenerator 140 generates a target trajectory for continuous traveling inthe current traveling lane without generating a target trajectory formaking the lane change. When the lane change has been approved by theoccupant, the action plan generator 140 repeatedly searches for thetarget position and an action plan for making the lane change isgenerated as soon as the target position can be set.

The suggestion of the lane change according to driving control (drivingassistance) described above may be made when the driving mode is mode Bor mode A. When the approval has been obtained for the lane changesuggestion, the driving mode is changed to mode C. When no approval hasbeen obtained, mode control for continuing the current mode B or mode Ais performed.

Mode D is a driving mode in which a certain degree of driving operationby the driver is required with respect to at least one of steering,acceleration, and deceleration of the vehicle M. In mode D, when aninstruction for causing the vehicle M to make a lane change has receivedfrom the driver according to an operation on a winker lever, drivingcontrol for making the lane change in an indicated direction isperformed. The lane change in modes B to D may be an example of a lanechange event.

In mode E, all types of steering, acceleration, and deceleration of thevehicle M are in a state of manual driving in which a driving operationby the driver is required. In both modes D and E, the task of monitoringthe front of the vehicle M is naturally imposed on the driver. Thedriving subject in modes C to E is the driver.

The mode determiner 150 determines the driving mode to be executed bythe vehicle M on the basis of the state of the driver. For example, themode determiner 150 may acquire the execution state of the task andchange the driving mode of the vehicle M to a driving mode in which thetask imposed on the driver is severer when the task associated with thedetermined driving mode is not executed by the driver. The modedeterminer 150 causes the HMI controller 170 to perform a controlprocess of prompting the driver to shift the driving mode to the manualdriving of mode E using the HMI 30, for example, when the driver is in aposture in which the driving mode cannot be shifted to manual driving inresponse to a request from the system (for example, when he or shecontinues to look outside of an allowable area or when a sign indicatingthat driving becomes difficult is detected) while mode A is beingexecuted. When there is no response from the driver even though aprescribed time period has elapsed after the HMI controller 170 wasallowed to execute a control process of prompting the driver to shiftthe driving mode to manual driving or when it is estimated that thedriver is not in a state in which manual driving is performed, the modedeterminer 150 performs a control process of causing the vehicle M to bestopped at a target position according to automated driving and stopping(ending) the automated driving after the vehicle M is stopped. After theautomated driving is stopped, the vehicle M is in the state of mode D orE and the vehicle M can be started according to the manual operation ofthe driver. Hereinafter, the same is true for “stopping of automateddriving.”

In mode B, when the driver is not monitoring the front, the modedeterminer 150 prompts the driver to monitor the surroundings using theHMI 30 and performs a control process of causing the vehicle M to bestopped at the target position and stopping the automated driving if thedriver does not respond. When the driver is not monitoring the front inmode C or is not gripping the steering wheel 82, the mode determiner 150prompts the driver to monitor the surroundings and/or grip the steeringwheel 82 using the HMI 30 and performs a control process of causing thevehicle M to be stopped at a target position and stopping automateddriving if the driver does not respond. In modes C and D, when the lanechange is not executed while the vehicle M reaches a prescribed point, acontrol process of causing the vehicle M to be stopped at the targetposition and stopping automated driving may be performed.

The driver state determiner 152 determines whether or not the driver isin a state suitable for driving. For example, the driver statedeterminer 152 monitors the state of the driver so that theabove-described mode is changed and determines whether or not the stateof the driver is a state according to the task. For example, the driverstate determiner 152 performs a posture estimation process by analyzingan image captured by the driver monitor camera 70 and determines whetheror not the driver is in a posture in which the driving mode cannot beshifted to manual driving in response to a request from the system. Thedriver state determiner 152 performs a line-of-sight estimation processby analyzing the image captured by the driver monitor camera 70 anddetermines whether or not the driver is monitoring the surroundings(more specifically, the front) of the vehicle M. When it is determinedthat the state is not a state according to the task for a prescribedtime period or more, the driver state determiner 152 determines that thedriver is in a state unsuitable for driving of the task. When it isdetermined that the state is a state according to the task, the driverstate determiner 152 determines that the driver is in a state suitablefor driving of the task. The driver state determiner 152 may determinewhether or not the driver is in a state in which the driving can bechanged.

Instead of (or in addition to) the driver's state described above, themode determiner 150 may determine the driving mode of the vehicle M onthe basis of the traveling state of the vehicle M, the presence orabsence of the driver's approval for the driving control to be executed,and the like. In this case, the mode determiner 150 determines thedriving mode of the vehicle M, for example, on the basis of adetermination result of the approval determiner 144. Details of thedetermination of the driving mode based on the above-describeddetermination result will be described below.

The mode change processor 154 performs various types of processes formaking a change from a current driving mode to the mode determined bythe mode determiner 150. For example, the mode change processor 154causes the HMI controller 170 to control the HMI 30 so that the driveris prompted to perform a prescribed operation or issues an instructionfor generating a target trajectory for executing or stopping theautomated driving. When the operator 45 has received an operation ofswitching the driving mode of the vehicle M to a prescribed mode, themode change processor 154 executes various types of processes forexecuting the mode. When the operator 45 has received the approval ofthe lane change to be made by the vehicle system 1, the approval of thecancellation of the lane change, or the like, the mode change processor154 executes various types of processes for switching to the modeassociated with content that has been received.

The second controller 160 controls the travel driving force outputdevice 200, the brake device 210, and the steering device 220 so thatthe vehicle M passes along the target trajectory generated by the actionplan generator 140 at the scheduled times.

The second controller 160 includes, for example, an acquirer 162, aspeed controller 164, and a steering controller 166. The acquirer 162acquires information of a target trajectory (trajectory points)generated by the action plan generator 140 and causes a memory (notshown) to store the information. The speed controller 164 controls thetravel driving force output device 200 or the brake device 210 on thebasis of a speed element associated with the target trajectory stored inthe memory. The steering controller 166 controls the steering device 220in accordance with a degree of bending of the target trajectory storedin the memory. The processes of the speed controller 164 and thesteering controller 166 are implemented by, for example, a combinationof feedforward control and feedback control. As an example, the steeringcontroller 166 executes feedforward control according to the curvatureof the road in front of the vehicle M and feedback control based on adeviation from the target trajectory in combination.

The HMI controller 170 notifies the occupant (the driver) of the vehicleM of prescribed information using the HMI 30. The prescribed informationincludes, for example, driving assistance information. For example, theHMI controller 170 may generate an image including the above-describedprescribed information and cause the display 32 to display the generatedimage or may generate speech indicating the prescribed information andcause the generated speech to be output from the speaker 34. The HMIcontroller 170 may output the information received by the HMI 30 and theoperator 45 to the communication device 20, the navigation device 50,the first controller 120, and the like. Details of the functions of theHMI controller 170 will be described below.

The travel driving force output device 200 outputs a travel drivingforce (torque) for enabling the vehicle M to travel to driving wheels.For example, the travel driving force output device 200 includes acombination of an internal combustion engine, an electric motor, atransmission, and the like, and an electronic control unit (ECU) thatcontrols the internal combustion engine, the electric motor, thetransmission, and the like. The ECU controls the above-describedcomponents in accordance with information input from the secondcontroller 160 or information input from the driving operation element80.

For example, the brake device 210 includes a brake caliper, a cylinderconfigured to transfer hydraulic pressure to the brake caliper, anelectric motor configured to generate hydraulic pressure in thecylinder, and a brake ECU. The brake ECU controls the electric motor inaccordance with the information input from the second controller 160 orthe information input from the driving operation element 80 so thatbrake torque according to a braking operation is output to each wheel.The brake device 210 may include a mechanism configured to transfer thehydraulic pressure generated according to an operation on the brakepedal included in the driving operation elements 80 to the cylinder viaa master cylinder as a backup. The brake device 210 is not limited tothe above-described configuration and may be an electronicallycontrolled hydraulic brake device configured to control an actuator inaccordance with information input from the second controller 160 andtransfer the hydraulic pressure of the master cylinder to the cylinder.

For example, the steering device 220 includes a steering ECU and anelectric motor. For example, the electric motor changes a direction ofsteerable wheels by applying a force to a rack and pinion mechanism. Thesteering ECU drives the electric motor in accordance with theinformation input from the second controller 160 or the informationinput from the steering wheel 82 of the driving operation element 80 tochange the direction of the steerable wheels.

[Lane Change Determiner, Approval Determiner, and HMI Controller]

Hereinafter, details of functions of the lane change determiner 142, theapproval determiner 144, and the HMI controller 170 will be described.FIG. 4 is a diagram for describing a lane change determiner 142, anapproval determiner 144, and an HMI controller 170. In the example ofFIG. 4, the vehicle M traveling at a speed VM in the lane L2 between twolanes L1 and L2 in which the vehicle M can travel in the same direction(the X-axis direction in FIG. 4) is shown. The lane L1 is partitioned bymarkings LL and CL and the lane L2 is partitioned by markings CL and RL.The marking CL is a marking indicating that the lane change can be madebetween the lanes L1 and L2. The lanes L1 and L2 are, for example, amotorway such as an expressway. In the following description, it isassumed that time t11 is earliest and times t12, t13, t14, t15, t16, andt17 are increasingly later than time t11 in that order. Although it isassumed below that the vehicle M is traveling in mode B at a time pointbefore time t11, the vehicle M may be traveling in mode A.

The lane change determiner 142 determines whether or not to cause thevehicle M to make the lane change according to driving control at thepresent time point or in the future on the basis of the surroundingsituation recognized by the recognizer 130. For example, the lane changedeterminer 142 determines whether or not to cause the vehicle M to makethe lane change on the basis of the traveling lane of the vehicle M andthe number of lanes of the road on which the vehicle M can travel in thesame direction including the traveling lane. The number of lanes may berecognized, for example, from the captured image of the camera 10 orfrom the map information (for example, the second map information 62).For example, the lane change determiner 142 compares a recommended lanedetermined by the recommended lane determiner 61 among a plurality oflanes in which the vehicle M can travel in the same direction with thetraveling lane of the vehicle M and determines to cause the vehicle M tomake the lane change when the recommended lane is not the same as thetraveling lane.

For example, when the vehicle M is traveling in a left lane of atwo-lane road and the recommended lane is a right lane thereof, the lanechange determiner 142 determines to cause the vehicle M to make the lanechange to the right lane. When the vehicle M travels in a right or leftlane of three lanes in the traveling direction and the recommended laneis a middle lane thereof, the lane change determiner 142 determines tocause the vehicle M to make the lane change to the middle lane. When adestination is preset by the navigation device 50 or the like, the lanechange determiner 142 determines to cause the vehicle M to make the lanechange to a lane in which the vehicle M can travel in a destinationdirection when the traveling lane of the vehicle M is not a lane inwhich the vehicle M can travel in the destination direction. The lanechange determiner 142 determines that it is necessary to make the lanechange to the recommended lane when the vehicle M is not traveling inthe preset recommended lane even if the vehicle is traveling on a roadhaving four or more lanes.

The lane change determiner 142 may compare the future recommended lanewith the traveling lane when a shape of a road on which the vehicle Mtravels changes in the future (for example, when there is a branchpoint, a merge point, or an increase or decrease in the number of lanesseveral kilometers ahead) and determine to cause the vehicle M to makethe lane change when the two lanes are different.

In the example of FIG. 4, it is assumed that the lane change determiner142 determines to cause the vehicle M to make the lane change from thelane L2 to the lane L1 at time t11 when the vehicle M has reached apoint P11. In this case, the HMI controller 170 generates an image forsuggesting (recommending) a lane change according to driving control,causes the display 32 to output the generated image, and provides animage notification to the driver.

FIG. 5 is a diagram showing an example of an image IM10 that suggests alane change. The image IM10 includes, for example, a suggestion contentdisplay area A11. In the suggestion content display area A11, forexample, information about the lane change suggestion is displayed. Inthe example of FIG. 5, in the suggestion content display area A11, textinformation such as “A lane change from the current traveling lane tothe right lane is recommended” and “Please perform an approval operationif you approve the lane change” is displayed. In addition to (or insteadof) the above-mentioned text information, the HMI controller 170 maydisplay an image (including an animation image) reminiscent of a lanechange for the occupant and cause an indicator lamp indicating a lanechange included in the HMI 30 to be turned on. In addition to (orinstead of) displaying an image or an indicator, the HMI controller 170may generate speech indicating the suggestion content (for example,speech corresponding to the text information shown in FIG. 5) and causethe generated speech to be output from the speaker 34 or cause a soundsuch as an alarm or a sound of a buzzer to be output from the speaker34. Thereby, the driver of the vehicle M is allowed to easily recognizethe fact that the lane change has been suggested from the vehicle system1 side and the operation content in the case of approval.

The approval determiner 144 determines whether or not the driver'sapproval for the above-described lane change suggestion has beenreceived. For example, the approval determiner 144 determines that thelane change according to the driving control has been approved when theoperator 45 has received a prescribed operation in a state in which theimage IM10 has been displayed on the display 32 by the HMI controller170. The prescribed operation may be, for example, an operation ofpressing the approval switch provided on the operator 45 once or anoperation of turning the approval switch on in a state in which theimage IM10 has been displayed. When the prescribed operation has notbeen received by the operator 45 within a prescribed time period, theapproval determiner 144 may determine that the approval has not beenreceived. When the approval has not been received, the lane change isnot made.

In the example of FIG. 4, it is assumed that the approval determiner 144determines that the driver's approval (lane change approval) for thelane change suggestion has been received at time t12 when the vehicle Mhas reached a point P12 after the image IM10 was displayed at time t11.In this case, the mode determiner 150 changes the driving mode of thevehicle M from mode B to mode C. After changing to mode C, the task ofgripping the steering as well as the task of monitoring the surroundingsis imposed on the driver. The action plan generator 140 searches for atarget position of a lane L1 that is a lane change destination on thebasis of the surrounding situation recognized by the recognizer 130,generates a target trajectory for causing a lane change for a targetposition to be made at a time point when the target position has beenfound in the search, and starts the lane change according to thegenerated target trajectory.

The action plan generator 140 generates an action plan for continuoustraveling in the lane L2 when the target position cannot be set due tothe presence of another vehicle traveling in the lane L1 or an obstaclesuch as a construction work area, and causes the vehicle M to travelaccording to the generated action plan. If the driver cannot make thelane change for a long time after approving the lane change, he or sheis likely to forget that he or she has approved the lane change. Thus,if the lane change is made after a while from the lane changeinstruction, there is a possibility that the driver will be surprisedand an unnecessary steering or brake operation will be performed becausethe lane change is made in a state in which the occupant is notexpecting the lane change. Therefore, the HMI controller 170 causes theHMI 30 to output information about the lane change when the lane changehas not been started by the driving controller even though the firstprescribed time period has elapsed after the lane change was approved.The term “the lane change has not been started” indicates that, forexample, the execution condition of the lane change is not satisfied.“The lane change has not been started” may indicate that the vehicle Mis entirely on the lane L2 side, that an amount of lateral movement tothe lane change destination (the lane L2 side) (an amount of lateralmovement of the road or an amount of movement in a —Y-axis direction inFIG. 4) is less than a prescribed amount, and that a steering angle ofthe vehicle M due to a lane change is less than a prescribed angle. Theinformation about the lane change includes, for example, informationabout the traveling state of the vehicle M related to the lane change orinformation about the cancellation of the lane change. The informationabout the cancellation of the lane change includes, for example, whetheror not it is necessary to cancel the approved lane change (whether ornot the cancellation is necessary), information about an operationmethod of canceling the lane change, and the like.

In the example of FIG. 4, the HMI controller 170 generates an imageshowing information about the lane change at time t13 when a firstprescribed time period (ΔT1) has elapsed (when the vehicle M has reacheda point P13) from time t12 and causes the display 32 to display thegenerated image. FIG. 6 is a diagram showing an example of an image IM20showing information about a lane change. The image IM20 includes, forexample, a traveling state display area A21. In the traveling statedisplay area A21, text information such as “The lane change still cannotbe started” and “The lane change can be canceled according to theapproval operation” is displayed as the information about the lanechange.

By displaying the image IM20 shown in FIG. 6, it is possible to allowthe driver to ascertain that the vehicle M is trying to make the lanechange according to the driving control, but the lane change cannot bestarted due to the surrounding situation or the like. Therefore, it ispossible to make it more difficult for the driver to forget that he orshe has approved the lane change according to the driving control (or itis possible to remind the driver of the fact). Further, it is possibleto limit an erroneous operation when the lane change is desired to becanceled by displaying information about the cancelation of the lanechange. When the HMI controller 170 notifies the driver of theinformation about the lane change, the HMI controller 170 provides thenotification to the driver only by displaying the image IM20 withoutproviding the notification by speech. Thereby, it is possible to reducean annoyance feeling given to the driver with respect to thenotification.

The approval determiner 144 determines whether or not the execution ofthe approved lane change according to the driving control has beencanceled (or whether or not the cancellation of the lane change has beenapproved) according to whether or not a prescribed operation has beenreceived by the operator 45 in a state in which the image IM20 has beendisplayed on the display 32 by the HMI controller 170. The time periodduring which the image IM20 has been displayed is a time period for thedriver to determine whether or not to continue the lane change. Forexample, in a state in which the image IM20 has been displayed, theprescribed operation may be an operation of pressing the cancelationswitch provided on the operator 45 once or an operation of pressing theapproval switch once or turning the approval switch off.

In FIG. 4, it is assumed that the approval determiner 144 determinesthat the lane change has been canceled (the cancelation thereof has beenapproved) at time t14 when the vehicle M has reached a point P14, forexample, in a state in which the image IM20 was displayed at time t13.In this case, the driving controller stops the lane change to the laneL1. The mode determiner 150 changes the driving mode from mode C to modeB and the driving control for traveling in the lane L2 is executed.

FIG. 7 is a diagram for describing a case where the lane change has notbeen canceled by the approval determiner 144. Because the processing upto time t13 in FIG. 7 is similar to the above-described processing, theprocessing after time t13 will be specifically described. At time t13,the HMI controller 170 causes the display of the image IM20 to end whena prescribed operation has not been received by the operator 45 eventhough the second prescribed time period has elapsed, for example, afterthe image IM20 was displayed on the display 32. The second prescribedtime period is shorter than the first prescribed time period. As shownin FIG. 7, the HMI controller 170 causes the display of the image IM20to end at time t15 when a second prescribed time period (ΔT2) haselapsed from time t13 when the image IM20 has been displayed (or at atime point when the vehicle M has reached a point P15). Thereby, it ispossible to ascertain that the vehicle M is trying to make the lanechange according to the driving control by displaying the image IM20 andreduce the annoyance given to the driver with respect to thenotification by ending the display of the image IM20 in a time periodshorter than the first prescribed time period. Because the lane changeaccording to the driving control is continued when the driver does notperform a lane change cancelation operation (a cancellation approvaloperation), the convenience of the driver can be improved as comparedwith a case where an operation of continuing the lane change isperformed every time the image IM20 is displayed.

Because the approval of the lane change is continued after time t13, thedriving controller causes the lane change to the lane L1 to be made ifthe execution condition of the lane change to the lane L1 is satisfied.

The HMI controller 170 may cause the display 32 to display the imageIM20 again at time t16 when a third prescribed time period (ΔT3) haselapsed (when the vehicle M has reached a point P16) after the displayof the image IM20 was ended at time t15. The third prescribed timeperiod is, for example, a time period equal to the first prescribed timeperiod. The third prescribed time period may be changed in accordancewith the number of times the image IM20 is displayed or a display timeperiod (a total display time period). For example, as the number ofnotifications to the driver or a notification time period according tothe display of the image IM20 increase, it is predicted that it will bedifficult for the occupant to forget that the lane change has beenapproved. Thus, the HMI controller 170 sets the third prescribed timeperiod to a time period longer than the first prescribed time period asthe number of times the image IM20 is displayed or the display timeperiod increases. The HMI controller 170 may perform control forcanceling the next display once according to the number of displayprocesses or the display time period of the image IM20. The thirdprescribed time period may be adjusted so that the third prescribed timeperiod becomes a longer time period when a traveling distance of thevehicle M is longer in a state in which the image IM20 has beendisplayed. In this way, by adjusting an interval at which the image IM20is displayed again, it is possible to reduce the annoyance caused by thedisplay of the image IM20. By repeatedly displaying the image IM20, itis possible to prevent the driver from forgetting that the lane changehas been approved, so that more appropriate driving assistance can beimplemented.

The HMI controller 170 causes the display of the image IM20 to end whena prescribed operation has not been received by the operator 45 even attime t17 when a fourth prescribed time period ΔT4 elapsed (a time pointwhen the vehicle M reached a point P17) after the image IM20 wasdisplayed at time t16. The fourth prescribed time period is a timeperiod equal to the second prescribed time period. The fourth prescribedtime period may be shortened in accordance with the number of times theimage IM20 is displayed and the display time period (a total displaytime period). However, the HMI controller 170 adjusts the fourthprescribed time period so that the fourth prescribed time period is notless than a predetermined minimum display time period. The HMIcontroller 170 causes the image IM20 to be displayed repeatedly on thebasis of the third prescribed time period and the fourth prescribed timeperiod when the lane change is not started even after time t17 and thecancellation of the lane change is not approved.

[Processing Flow]

FIG. 8 is a flowchart showing an example of a flow of a process executedby the automated driving controller 100 of the embodiment. Hereinafter,processes related to the suggestion of the lane change from the vehiclesystem 1 side, the approval of the suggestion, and the cancellation ofthe lane change among the processes executed by the automated drivingcontroller 100 will be mainly described. It is assumed that the vehicleM is traveling in mode B when the following process is executed. In thefollowing process, it is assumed that the driver's state determined bythe driver state determiner 152 is a state corresponding to a taskimposed on the driver in the driving mode executed by the vehicle M.Therefore, the mode according to the state of the driver is not changedin the process of FIG. 8.

In the example of FIG. 8, the recognizer 130 recognizes a surroundingsituation of the vehicle M (step S100). Subsequently, the lane changedeterminer 142 determines whether or not to suggest a lane change of thevehicle M on the basis of the recognized surrounding situation of thevehicle M or the like (step S102). When it is determined to suggest thelane change, the HMI controller 170 generates an image (for example, theimage IM10) showing the lane change suggestion and causes the display 32to output the generated image (step S104).

Subsequently, the approval determiner 144 determines whether or not theapproval of the suggested lane change has been received by the operator45 (step S106). When it is determined that the approval of the lanechange has been received, the mode determiner 150 determines to switchthe driving mode of the vehicle M from mode B to mode C (step S108).Thereby, the driving controller switches the mode from mode B to mode C.Subsequently, the action plan generator 140 determines whether or notthe lane change to the lane change destination can be made (step S110).When it is determined that the lane change can be made, the drivingcontroller causes the lane change to the lane change destination to bemade (step S112). Subsequently, the mode determiner 150 determines tochange the driving mode of the vehicle M from mode C to mode B (stepS114). Thereby, the mode is switched to mode B by the drivingcontroller.

When the lane change cannot be made after the approval of the lanechange (in other words, when the lane change has not been started) inthe processing of step S110, the action plan generator 140 determineswhether or not the first prescribed time period has elapsed after thereception of the approval of the lane change (step S116). When it isdetermined that the first prescribed time period has not elapsed, theprocess returns to the processing of step S110. When it is determinedthat the first prescribed time period has elapsed, the HMI controller170 generates an image (for example, the image IM20) includinginformation about the cancelation of the lane change and causes thedisplay to display the generated image (step S118).

Subsequently, the HMI controller 170 determines whether or not theapproval for canceling the lane change has been received (step S120).When it is determined that the cancellation approval has been received,the driving controller executes the processing of step S114 (a processof switching the driving mode from mode C to mode B) without causing thelane change to be made. When it is determined that the cancellationapproval has not been received, the HMI controller 170 determineswhether or not the second prescribed time period has elapsed after theimage including the information about the cancellation of the lanechange was displayed on the display 32 (step S122). When it isdetermined that the second prescribed time period has not elapsed, theprocess returns to the processing of step S120. When it is determinedthat the second prescribed time period has elapsed, the HMI controller170 ends the display of the image including the information about thecancellation of the lane change (step S124) and returns to theprocessing of step S110. Thereby, the process of the present flowchartis executed. When it is determined that the lane change has not beensuggested in the processing of step S102 or when it is determined thatthe approval of the lane change has not been received by the operator 45in the processing of step S106, the process of the present flowchartends.

Modified Examples

In the above-described embodiment, although control is performed when aprescribed operation has been received by the operator 45 after theimage IM10 and the image IM20 was displayed, an approval determinationprocess or the like may be performed by displaying a graphical userinterface (GUI) switch for selecting whether or not to approve (orcancel) the lane change in the image IM10 and the image IM20 andreceiving an operation of the driver on the displayed GUI switch insteadof the control. In this case, the display 32 (the touch panel) on whichan image including the GUI switch is displayed is an example of a“receiver.”

According to the above-described embodiment, a driving assistance deviceincludes the recognizer 130 configured to recognize a surroundingsituation of the vehicle M; a driving controller configured to controlat least steering of the vehicle M on the basis of the surroundingsituation recognized by the recognizer 130; the operator (an example ofa receiver) 45 configured to receive an operation of an occupant of thevehicle M; and the HMI controller (an example of an output controller)170 configured to cause the HMI 30 (an example of an output) to output atraveling state of the vehicle M, wherein the driving controller causesa lane change of the vehicle M to be made when the operator 45 receivesan instruction for allowing the vehicle M to make the lane change and acondition in which the lane change is made is satisfied, and wherein theHMI controller 170 causes the HMI 30 to output information about thelane change when the lane change has not been started by the drivingcontroller even though a first prescribed time period has elapsed afterthe operator 45 received the instruction, so that more appropriatedriving assistance can be performed.

Specifically, according to the above-described embodiment, when aprescribed time period has elapsed without the lane change being madeafter the approval of the lane change (after the reservation), it ispossible to prevent the driver from forgetting that he or she hasapproved the lane change according to the elapse of time by providing anotification of information about the lane change and to prevent thedriver from being surprised by the execution of the lane change.According to the embodiment, it is possible to improve the convenienceof the driver because the lane change is not automatically canceledunless the cancelation operation is performed by including informationabout whether or not the cancelation of the lane change is required or acancelation method in information about the lane change whosenotification is provided to the occupant. According to the embodiment,the annoyance given to the driver with respect to the notification canbe reduced by shortening the display time period of the informationabout the lane change and only displaying an image for the informationabout the lane change on the display 32.

The embodiment described above can be represented as follows.

An assistance control device including:

a storage device storing a program; and

a hardware processor,

wherein the hardware processor executes the program stored in thestorage device to:

recognize a surrounding situation of a vehicle;

control at least steering of the vehicle on the basis of the recognizedsurrounding situation in a driving controller;

receive an operation of an occupant of the vehicle;

cause an output to output a traveling state of the vehicle;

cause a lane change of the vehicle to be made when an instruction forallowing the vehicle to make the lane change is received and a conditionin which the lane change is made is satisfied; and

cause the output to output information about the lane change when thelane change has not been started by the driving controller even though afirst prescribed time period has elapsed after the instruction wasreceived.

While preferred embodiments of the invention have been described andillustrated above, it should be understood that these are exemplary ofthe invention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. A driving assistance device comprising: arecognizer configured to recognize a surrounding situation of a vehicle;a driving controller configured to control at least steering of thevehicle on the basis of the surrounding situation recognized by therecognizer; a receiver configured to receive an operation of an occupantof the vehicle; and an output controller configured to cause an outputto output a traveling state of the vehicle, wherein the drivingcontroller causes a lane change of the vehicle to be made when areceiver receives an instruction for allowing the vehicle to make thelane change and a condition in which the lane change is made issatisfied, and wherein the output controller causes the output to outputinformation about the lane change when the lane change has not beenstarted by the driving controller even though a first prescribed timeperiod has elapsed after the receiver received the instruction.
 2. Thedriving assistance device according to claim 1, wherein the informationabout the lane change includes information about cancelation of the lanechange.
 3. The driving assistance device according to claim 1, whereinthe output controller causes a process of outputting the information toend when the receiver has not received an instruction for canceling thelane change even though a second prescribed time period has elapsedafter the information about the lane change was output to the output,and wherein the driving controller causes a process of making the lanechange to continue.
 4. The driving assistance device according to claim3, wherein the second prescribed time period is shorter than the firstprescribed time period.
 5. The driving assistance device according toclaim 1, wherein the output includes a display, and wherein the outputcontroller causes the display to display an image showing informationabout the lane change.
 6. The driving assistance device according toclaim 3, wherein the output controller causes the output to outputinformation about the lane change when the lane change has not beenstarted by the driving controller even though a third prescribed timeperiod has elapsed after the process of outputting the information wasended.
 7. The driving assistance device according to claim 6, whereinthe third prescribed time period is a time period equal to the firstprescribed time period or a time period that is changed in accordancewith the number of times the information about the lane change has beenoutput or the time when the information about the lane change has beenoutput.
 8. A driving assistance method comprising: recognizing, by acomputer, a surrounding situation of a vehicle; controlling, by thecomputer, at least steering of the vehicle on the basis of therecognized surrounding situation in a driving controller; receiving, bythe computer, an operation of an occupant of the vehicle; causing, bythe computer, an output to output a traveling state of the vehicle;causing, by the computer, a lane change of the vehicle to be made whenan instruction for allowing the vehicle to make the lane change isreceived and a condition in which the lane change is made is satisfied;and causing, by the computer, the output to output information about thelane change when the lane change has not been started by the drivingcontroller even though a first prescribed time period has elapsed afterthe instruction was received.
 9. A computer-readable non-transitorystorage medium storing a program for causing a computer to: recognize asurrounding situation of a vehicle; control at least steering of thevehicle on the basis of the recognized surrounding situation in adriving controller; receive an operation of an occupant of the vehicle;cause an output to output a traveling state of the vehicle; cause a lanechange of the vehicle to be made when an instruction for allowing thevehicle to make the lane change is received and a condition in which thelane change is made is satisfied; and cause the output to outputinformation about the lane change when the lane change has not beenstarted by the driving controller even though a first prescribed timeperiod has elapsed after the instruction was received.